Mochi: Visual Log-Analysis Based Tools for Debugging Hadoop (CMU-PDL-09-103)

Mochi, a new visual, log-analysis based debugging tool correlates Hadoop’s behavior in space, time and volume, and extracts a causal, unified controland data-flow model of Hadoop across the nodes of a cluster. Mochi’s analysis produces visualizations of Hadoop’s behavior using which users can reason about and debug performance issues. We provide examples of Mochi’s value in revealing a Hadoop job’s structure, in optimizing real-world workloads, and in identifying anomalous Hadoop behavior, on the Yahoo! M45 Hadoop cluster. Acknowledgements: The authors would like to acknowledge Christos Faloutsos and U Kang for discussions on the HADI Hadoop workload and for providing log data. This research was partially funded by the Defence Science & Technology Agency, Singapore, via the DSTA Overseas Scholarship, and sponsored in part by the National Science Foundation, via CAREER grant CCR-0238381 and grant CNS-0326453.